Update to 1.12.0 #137

andrewdnolan · 2025-07-23T20:26:19Z

⚠️ Work in Progress: This release is not planned until November 2025. Some preliminary work was done in order to familiarize @andrewdnolan with the e3sm-unified configuration and deployment process. Release candidates have been tested on chrysalis, but have not been uploaded to conda-forge.

In this version

Many packages have been updated, including:
- chemdyg 1.1.1
- e3sm_diags 3.1.0
- e3sm_to_cmip 1.13.0
- livvkit 3.2.0
- mache 2.0.0
- esmf 8.9.0
- esmfpy 8.9.0
- hdf5 1.14.6
- mache 1.31.0
- mpas_analysis 1.14.0
- mpas_tools 1.4.0
- nco 5.3.6
- pcmdi_metrics 4.0.0
- tempest-extremes 2.4
- xcdat 0.10.1
- zppy 3.1.0
- zppy-interfaces 0.2.0
- zstash 1.5.0
- dask 2025.9.1
- matplotlib 3.10.6
- proj 9.6.2
- pyproj 3.7.2
- xarray 2025.9.0
python versions 3.11, 3.12, and 3.13 will be supported

New and improved

TBD

Deployment

Deployed on:

Sync Diagnostics

Synced on:

Delete old test envs

recipes/e3sm-unified/meta.yaml

xylar · 2025-07-24T09:37:55Z

@andrewdnolan, one thing we need to figure out is how to set permissions on the base conda environment and its contents such that multiple maintainers can create environments but users can't install packages into the environment by mistake.

My thought here is that it might be safer to introduce a conda directory similar to the spack directory and to have fresh conda installations under e3sm_unified_1_12_0, etc. So what was base now becomes conda/e3sm_unified_1_12_0 or whatever. This way, we could set permissions such that conda is group writable but e3sm_unified_1_12_0, etc. and their contents are only writable by one maintainer.

Could you look into this? The relevant code to fix would be:
https://github.com/E3SM-Project/e3sm-unified/blob/main/e3sm_supported_machines/shared.py#L132-L133
The function would need to take the version number and convert it to the desired syntax with underscores, similar to what is done for spack path names here:
https://github.com/E3SM-Project/e3sm-unified/blob/main/e3sm_supported_machines/bootstrap.py#L112-L117
But we don't need the machine, compiler or MPI variant because the conda base environment doesn't care about those things.

Changes default version of python to 3.13 and release versions to be 3.11, 3.12, and 3.13. To support the new python versions all CDAT related packages have been removed from the `meta.yaml` and the associated testing of `cdms2` in the `bootstrap.py` file. The `pcmdi_metrics` package is currently commented out of the `meta.yaml` because it has `genutils`, `cdutils`, `cdms2`, and `cdp` as dependencies. When a new version of `pcmdi_metrics`, without CDAT dependencies, is released it will be uncommented out of the `meta.yaml`. With the update to python 3.13 the build string for the python version has changed and is now dealt with accordingly.

A cmake based build system was introduce in version 2.3.0. The spack package recipe for TempestExtremes uses a cmake build system, so a version >=2.3.0 is now neeed

Also simplified the spack environment name given that the spack installation lives within a versioned e3sm-unified direcotry.

recipes/e3sm-unified/meta.yaml

andrewdnolan · 2025-09-30T22:12:58Z

e3sm_supported_machines/bootstrap.py

    base_path = config.get('e3sm_unified', 'base_path')
-    spack_base = f'{base_path}/{env_name}/{machine}/'
+    spack_base_path = (
+        f'{base_path}/{env_name}/{machine}/spack/spack_for_{compiler}_{mpi}'


With this addition the directory layout(on chrysalis) would look like:

$ tree -L 4 /lcrc/soft/climate/e3sm-unified/ /lcrc/soft/climate/e3sm-unified/ ├── e3sm_unified_x_x_x │ └── chrysalis │ ├── base │ ├── spack │ │ └── spack_for_gnu_openmpi

for e3sm-unified where there's only ever on compiler and mpi combination, maybe the intermidate spack folder is unneeded. Happy to defer to you about that @xylar.

Yeah, the spack folder is redundant for now but consistent with future plans, right? Let's keep it for consistency.

recipes/e3sm-unified/meta.yaml

xylar · 2025-10-02T12:53:49Z

recipes/e3sm-unified/meta.yaml

    - jupyter
    - livvkit 3.1.0
-    - mache 1.28.0
+    - mache 1.31.0


During deployment, please use the branch:
https://github.com/E3SM-Project/mache/tree/update-to-2.0.0
with the --mache_fork and --mache_branch flags.

Feel free to update that branch (which includes E3SM-Project/mache#303) as needed to get things working.

We can make a release candidate when we know that things are working but right now it feels premature.

xylar · 2025-10-02T13:19:06Z

e3sm_supported_machines/default.cfg

 # the version of mache to use during deployment (should match the version used
 # in the package itself)
-mache = 1.28.0
+mache = 1.31.0


Can leave this for now but will eventually need to be 2.0.0rcX once we have a release candidate.

e3sm_supported_machines/default.cfg

Update package version to be in sync with confluence page Co-authored-by: Xylar Asay-Davis <[email protected]>

Remove constraint on notebook which was for compatability with nb_conda, which has been removed. notebook has been left in, without constraint, so that jupyter notebooks are still avaialable

recipes/e3sm-unified/meta.yaml

xylar · 2025-10-15T09:43:31Z

e3sm_supported_machines/deploy_e3sm_unified.py

e3sm_supported_machines/shared.py

This is necessary to avoid a SbangPathError in spack on Frontier. This merge also drops the requirement that the machine be specified. There is no longer an attempt to use a shared conda base environment as the conda environment for bootstrapping. Instead, the person deploying Unified must supply the `--conda` flag or have a base environment loaded before deploying.

The deploy and bootstrap scripts are now getting totally different conda bases in totally different ways so a shared function doesn't make sense anymore. This merge splits the functionality back into these 2 scripts for better clarity and simplicity.

Now consitent with the new layout for e3sm-unified

This is needed to unload a problem module on ALCF Polaris and may be useful on other machines in the future. This requires a new capability in mache.

andrewdnolan · 2025-10-27T17:15:25Z

@xylar would mind looking over the most recent commit?

Assuming you're OK with the approach/layout here, I'll get cranking on the NCO PR.

xylar · 2025-10-27T17:57:12Z

@andrewdnolan, as discussed on slack, looks good!

recipes/e3sm-unified/meta.yaml

Co-authored-by: Xylar Asay-Davis <[email protected]>

Adds moab and mache RCs

xylar reviewed Jul 24, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

andrewdnolan added 6 commits September 18, 2025 09:00

Update to v1.12.0rc1

739d2a6

Update deployment for mache 1.31.0

e398e69

Clean up meta.yaml

1c6a9f3

Update TempestExtremes to v2.3.1

c5deaec

A cmake based build system was introduce in version 2.3.0. The spack package recipe for TempestExtremes uses a cmake build system, so a version >=2.3.0 is now neeed

Move conda installation to versioned directory

64eb787

Also simplified the spack environment name given that the spack installation lives within a versioned e3sm-unified direcotry.

andrewdnolan force-pushed the update-to-1.12.0 branch from d9080aa to 64eb787 Compare September 18, 2025 14:44

xylar reviewed Sep 30, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

Put compiler and mpi in spack directory name

72dce2d

andrewdnolan commented Sep 30, 2025

View reviewed changes

xylar reviewed Oct 1, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

xylar reviewed Oct 2, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

xylar reviewed Oct 2, 2025

View reviewed changes

e3sm_supported_machines/default.cfg Outdated Show resolved Hide resolved

xylar reviewed Oct 2, 2025

View reviewed changes

e3sm_supported_machines/default.cfg Outdated Show resolved Hide resolved

xylar reviewed Oct 2, 2025

View reviewed changes

e3sm_supported_machines/default.cfg Outdated Show resolved Hide resolved

andrewdnolan and others added 6 commits October 6, 2025 10:34

Apply suggestions from code review

188973e

Update package version to be in sync with confluence page Co-authored-by: Xylar Asay-Davis <[email protected]>

Manually apply more suggestions from code review

4b5dc34

Explicitly add notebook as dependecy

a4303a1

Remove constraint on notebook which was for compatability with nb_conda, which has been removed. notebook has been left in, without constraint, so that jupyter notebooks are still avaialable

Add zppy-interfaces dev label

6c4ef8d

Remove testing command for nb_conda

e8696ac

Sync meta.yaml with confluence page

c31915c

xylar reviewed Oct 14, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

xylar and others added 2 commits October 16, 2025 13:37

Add "Verify compute-node activation" step to deployment docs

7113180

If depolying a RC add the dev label to channels

047c730

andrewdnolan added 4 commits October 20, 2025 09:44

fix return variables name

c0f810e

Bump mpi4py version

4f314d9

uncomment file permsission updates

2cc66fa

Update to v1.12.0rc2

1f21059

andrewdnolan force-pushed the update-to-1.12.0 branch from 3c0c625 to 1f21059 Compare October 20, 2025 15:33

xylar reviewed Oct 22, 2025

View reviewed changes

e3sm_supported_machines/deploy_e3sm_unified.py Outdated Show resolved Hide resolved

xylar reviewed Oct 22, 2025

View reviewed changes

e3sm_supported_machines/shared.py Outdated Show resolved Hide resolved

xylar and others added 4 commits October 23, 2025 11:09

Clean up get_conda_base()

d6fdb3a

The deploy and bootstrap scripts are now getting totally different conda bases in totally different ways so a shared function doesn't make sense anymore. This merge splits the functionality back into these 2 scripts for better clarity and simplicity.

Update paths the need permission changes

5f9abfe

Now consitent with the new layout for e3sm-unified

Start user restricted perms at machine dir level

14d1df4

andrewdnolan force-pushed the update-to-1.12.0 branch from 458a68e to 14d1df4 Compare October 23, 2025 16:03

Add pre_conda_script to load scripts

ced1fb5

This is needed to unload a problem module on ALCF Polaris and may be useful on other machines in the future. This requires a new capability in mache.

xylar force-pushed the update-to-1.12.0 branch from b8c5960 to ced1fb5 Compare October 24, 2025 10:54

Update symlinks created for NCO

24ebabe

andrewdnolan added 3 commits October 29, 2025 09:43

Update to v1.12.0rc3

c0ae19f

Update version number in meta.yaml

fef77bf

Add check for ${PBS_JOBID} to templates

cfc804c

xylar reviewed Nov 5, 2025

View reviewed changes

recipes/e3sm-unified/meta.yaml Outdated Show resolved Hide resolved

andrewdnolan and others added 4 commits November 5, 2025 09:27

Update recipes/e3sm-unified/meta.yaml

40b30ac

Co-authored-by: Xylar Asay-Davis <[email protected]>

Update to 1.12.0rc4

fec404f

Adds moab and mache RCs

Ensure prebuilt wheel for mpi4py is not installed

851c1d9

Update base path permission non recursively

577f929

andrewdnolan force-pushed the update-to-1.12.0 branch from 2d2f1a1 to 577f929 Compare November 5, 2025 21:30

Update to 1.12.0 #137

Are you sure you want to change the base?

Update to 1.12.0 #137

Uh oh!

Conversation

andrewdnolan commented Jul 23, 2025 • edited by xylar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

In this version

New and improved

Deployment

Sync Diagnostics

Delete old test envs

Uh oh!

Uh oh!

xylar commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewdnolan Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

xylar Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xylar Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

xylar Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xylar commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Group write permission

Uh oh!

Uh oh!

Uh oh!

andrewdnolan commented Oct 27, 2025

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andrewdnolan commented Jul 23, 2025 •

edited by xylar

Loading

xylar commented Jul 24, 2025 •

edited

Loading

xylar commented Oct 15, 2025 •

edited

Loading